A probabilistic segment model combining lexical cohesion and disruption for topic segmentation (Un modèle segmental probabiliste combinant cohésion lexicale et rupture lexicale pour la segmentation thématique) [in French]

A probabilistic segment model combining lexical cohesion and disruption for topic segmentation Identifying topical structure in any text-like data is a challenging task. Most existing techniques rely either on maximizing a measure of the lexical cohesion or on detecting lexical disruptions. A novel method combining the two criteria so as to obtain the best trade-off between cohesion and disruption is proposed in this paper. A new statistical model is defined, based on the work of Isahara and Utiyama (2001), maintaining the properties of domain independence and limited a priori of the latter. Evaluations are performed both on written texts and on automatic transcripts of TV shows, the latter not respecting the norms of written texts, thus increasing the difficulty of the task. Experimental results demonstrate the relevance of combining lexical cohesion and disrupture. MOTS-CLÉS : segmentation thématique, cohésion lexicale, rupture de cohésion, journaux télévisés.